Skip to content

fix(hl7): make policy safe-harbor actually Safe Harbor for the encounter loci - #49

Merged
NSchatz merged 6 commits into
mainfrom
deid-encounter-loci
Aug 11, 2026
Merged

fix(hl7): make policy safe-harbor actually Safe Harbor for the encounter loci#49
NSchatz merged 6 commits into
mainfrom
deid-encounter-loci

Conversation

@NSchatz

@NSchatz NSchatz commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

The defect, reproduced by probe

Under policy: "safe-harbor", seven identifying loci survived byte-identical with no manifest entry at all, because RETAIN_SEGMENTS retains a segment whole and nothing carved its identifying fields back out:

PV1-19 visit number, PV1-44 admit, PV1-45 discharge, OBR-7 observation date, DG1-5 diagnosis date, and the placer/filler order numbers (OBR-2/3, ORC-2/3).

The manifest read only PID-3[0]:transformed PID-5:removed PID-7:transformed. Reproduced on f8b7f56 before any change, and again as a red test.

HIPAA Safe Harbor 45 CFR 164.514(b)(2)(i)(C) requires removal of all elements of dates except year that are directly related to an individual, and names admission and discharge dates in the regulation text itself; a visit or order number is a unique identifying code the (R) catch-all reaches. A policy named safe-harbor that returns them is a compliance trap for anyone who trusts the name. Founder decision, 2026-08-11; breaking on purpose while the package is pre-alpha, since 0.1.0 would freeze the guarantee.

What changes

  • SAFE_HARBOR_PROFILE retains nothing. The four dates generalize to their year; the five identifier loci are removed as category (R).
  • LIMITED_DATA_SET_PROFILE keeps all seven, grounded in 164.514(e)(2), whose limited-data-set exclusion list enumerates sixteen direct identifiers, contains no date, and has no catch-all. Encoded as data (LIMITED_DATA_SET_DIRECT_IDENTIFIERS, isRetainableCategory()) so the citation is enforced rather than restated: exactly two of the eighteen categories are retainable.
  • Retention takes three independent keys, and a missing one always means the transform runs: the adapter proposes a class, the configured options must list that class, and the resolved category must be retainable. So an adapter cannot retain anything by itself, and an options bag that omits retainedLoci keeps nothing.
  • PV1-19 is routed by its CX-5 identifier-type code, exactly as PID-3 is. This is load bearing, not cosmetic: a visit-number field routinely carries a medical record or account number, both named by 164.514(e)(2). An MR-typed value now gets the same keyed surrogate as the matching PID-3 entry instead of being republished in the clear beside it.
  • A safe-harbor-labelled policy may not retain at all, whatever the options bag says: fatal DEID_POLICY_INVALID, the retention analogue of the existing date-shift label guard.
  • Anything still retained is recorded: disposition retained, transform retain, code DEID_RESIDUAL_RETAINED, so it reaches the Expert-Determination support report's residual inventory instead of being invisible in both artifacts. Each inventory row now carries its transform, so a kept year is distinguishable from a kept full-precision timestamp.
  • defineDeidProfile()'s widen-never-narrow contract now covers retention, and it reads the opposite way round from a transform override: dropping a retained class removes more (allowed); adding one keeps more (fatal). A subset test, not a rank comparison.

The leak corpus was structurally blind to this class, which is the deeper defect

No HL7 fixture carried PV1-19/44/45 at all, so the headline gate could not have caught a regression: a sweep only reports on sentinels a fixture actually seeds. test/fixtures/hl7/adt-a03.hl7 seeds all seven plus the usual demographics and is wired into test/corpus/leak-corpus.test.ts.

Proven red on the pre-fix source. The new fixture and tests run against f8b7f56's src/ fail 8 tests, including the headline leak gate reporting all seven encounter sentinels surviving. The refuter independently reproduced this.

Positive control in both directions, because a detector that reports zero can be a gap rather than a clearance: safe-harbor removes all seven; limited-data-set still carries all seven byte-identical, composites and assigning authorities intact, while still removing the patient identifiers 164.514(e)(2) does name.

Corrected claims

docs-content/limitations.md is the honesty page, and its "loci absent from the parser models fail closed" row read as the opposite of the truth for a retained segment. That claim is deleted, not caveated. In its place the page states the true class: every field of a retained segment the carve-out does not name is still passed through and recorded nowhere, still including full-precision timestamps in EVN, PV2, PR1, RXA, RXD, FT1, TXA and SPM and the provider names in PV1-7/8 and OBR-16. The carve-out narrows that class; it does not close it.

Gate

scripts/verify.sh deid green: 621 tests, typecheck, lint, format:check, check, phi-scan, check-no-internal-refs, coverage, build, attw, smoke.

conformance-refuter pass 1 returned REFUTED with three INTRODUCED findings inside the bar, all addressed in a1851a1:

  • F2 (blocker) the 164.514(e)(2) ground was asserted rather than checked, so an MR-typed PV1-19 was retained under limited-data-set, republishing in the clear the MRN pseudonymized at PID-3 in the same message.
  • F3 the engine retained on the locus marker alone and never read options.retainedLoci, making the documented fail-closed default false at the layer it was declared on.
  • F1 the corrected honesty page had replaced a true broad limitation with a false narrow enumeration, which is precisely what the acceptance criteria test for.

Two findings were tagged PRE-EXISTING and are deliberately not fixed here; they are going out as their own backlog lines.

Note on history

Two commits in this branch's history are titled park: UNVERIFIED, UNGRADED .... They were written by a coordinator to preserve work in progress after the worker fleet died mid-slice on an auth error. Everything in them has since been re-measured from scratch, verified and graded. Squash-merging collapses them.

…esidual

Parked by the coordinator after every worker in the fleet died on an API auth
error (Login expired) mid-slice. This commit exists ONLY so the work is not
lost. It MUST NOT be treated as landed:

  - scripts/verify.sh NEVER RAN on it
  - the conformance-refuter NEVER GRADED it
  - it changes what leaves the box in a de-identification toolkit, which is
    the highest-consequence surface in this tree

Intended scope was the founder decision of 2026-08-11: make policy safe-harbor
actually Safe Harbor by dropping PV1-19, PV1-44, PV1-45, OBR-7 and DG1-5 from
retention under SAFE_HARBOR_PROFILE while keeping them under
LIMITED_DATA_SET_PROFILE, plus seeding the leak corpus that is structurally
blind to the class and correcting docs-content/limitations.md.

Whether this tree does any of that is UNKNOWN and unmeasured. src/retention.ts
and test/fixtures/hl7/adt-a03.hl7 are new and ungraded; probe.local.mts is
worker scratch. Re-measure from scratch before building on it. Do not
cherry-pick it green.
119 lines the worker flushed to test/corpus/leak-corpus.test.ts after the first
park commit. Same status as everything else on this branch: verify never ran,
no refuter graded it.

Worth keeping specifically because seeding this corpus was the DEEPER half of
the slice. The gate was structurally blind to the class its own roadmap says it
must catch -- no HL7 fixture carried PV1-19/44/45 -- so a fix without a fixture
would leave nothing to catch a regression.
…loci (DEID-SAFE-HARBOR-DATES)

Founder decision, 2026-08-11, on a measured finding: under `policy: "safe-harbor"` seven
identifying loci survived BYTE-IDENTICAL with no manifest entry at all, because
RETAIN_SEGMENTS retains a segment whole and nothing carved its identifying fields back out:

  PV1-19 visit number, PV1-44 admit, PV1-45 discharge, OBR-7 observation date,
  DG1-5 diagnosis date, and the placer/filler order numbers (OBR-2/3, ORC-2/3).

The manifest read only `PID-3[0]:transformed PID-5:removed PID-7:transformed`.

HIPAA Safe Harbor 164.514(b)(2)(i)(C) requires removal of all elements of dates except year
directly related to an individual, and names admission and discharge dates in the regulation
text itself; the visit and order numbers are unique identifying codes the (R) catch-all
reaches. A policy NAMED safe-harbor that returns them is a compliance trap for anyone who
trusts the name.

WHAT CHANGES

- New format-agnostic retention model (src/retention.ts): two named classes,
  `encounter-dates` and `encounter-identifiers`, each grounded in a DIFFERENT regulation.
  164.514(e)(2)'s limited-data-set exclusion list enumerates sixteen direct identifiers and
  contains no date and NO CATCH-ALL, which is exactly why these two classes may stay in a
  limited data set and may not stay under Safe Harbor.
- SAFE_HARBOR_PROFILE retains neither class; LIMITED_DATA_SET_PROFILE retains both.
- New HL7 carve-out table (RETAINED_LOCUS_RULES): retaining a SEGMENT no longer means
  retaining every field in it. Under safe-harbor the four dates generalize to their year and
  the five identifier loci are blocked as (R).
- Anything still retained is RECORDED. A kept locus emits a manifest entry with
  disposition `retained`, transform `retain`, code DEID_RESIDUAL_RETAINED, so it reaches the
  Expert-Determination support report's residual inventory instead of being invisible twice
  over. DeidManifestEntry.disposition and ReportDisposition gain "retained";
  DispositionSummary gains a `retained` count.
- widen-never-narrow now covers retention, and it reads the OPPOSITE way round from a
  transform override: DROPPING a class removes more (allowed), ADDING one keeps more
  (fatal DEID_PROFILE_INVALID). It is a subset test, not a rank comparison.
- Fail closed by default: retention rides on DeidOptions and only profileOptions() puts it
  there, so a bare options bag, or one built by hand from a profile's `policy`, keeps
  nothing. The engine's retention branch also runs AFTER the clinical / free-text /
  unrecognized-structure guards, so the flag can never keep prose or unknown structure.
- New applier edit kind `none`: a retained locus is not written back at all, so it survives
  byte-identical rather than being flattened into a single-component field.

THE LEAK CORPUS WAS STRUCTURALLY BLIND TO THIS CLASS, WHICH IS THE DEEPER DEFECT

No HL7 fixture carried PV1-19/44/45 at all, so the headline gate could not have caught a
regression: a sweep only reports on sentinels a fixture actually seeds. test/fixtures/hl7/adt-a03.hl7
seeds all seven plus the usual demographics, and is wired into test/corpus/leak-corpus.test.ts.

Proven RED on the pre-fix source: the new fixture + tests over f8b7f56's src/ fail 8 tests,
including the headline leak gate reporting all seven encounter sentinels surviving.

POSITIVE CONTROL IN BOTH DIRECTIONS, because a detector that reports zero can be a gap rather
than a clearance: safe-harbor removes all seven, limited-data-set still carries all seven
byte-identical (composites intact, assigning authority and all) while still removing the
patient identifiers 164.514(e)(2) does name.

DOCS

docs-content/limitations.md's "loci absent from the parser models fail closed" row read as the
opposite of the truth for a retained segment, so it is DELETED rather than caveated. Its place
is taken by an accurate section that names what is retained AND names what is still passed
through and recorded nowhere: SPM-17, PV1-7/8, OBR-16. The support report and its guide no
longer redirect a determiner to "each format's retained-segment notes" for something the
report can now inventory.
…angeset

`retain` joins the fail-closed arm of applyTransform explicitly rather than
falling into `default`, so the exhaustiveness rule is satisfied by naming it and
the comment records WHY reaching that arm must still block: it means a policy
asked for retention per-category, which is not how retention is decided.

verify.sh is green on this tree: typecheck, lint, format:check, check,
phi-scan, check:no-internal-refs, test:coverage, build, attw, smoke.
…er pass 1

Pass 1 returned VERDICT: REFUTED with three INTRODUCED findings inside the bar.
All three are addressed here; the remedy diff is what pass 2 grades.

F2 (blocker, criteria 2 and 3) -- the 164.514(e)(2) ground was ASSERTED, not
checked. PV1-19 was hard-coded to the (R) catch-all without ever reading CX-5,
although the adapter already ships categoryForIdentifierType() and uses it for
PID-3. 164.514(e)(2)(vii) NAMES medical record numbers and (ix) NAMES account
numbers, so for a routine `MR`-typed visit number the slice's own justification
was false -- and under limited-data-set the pass republished in the clear the
very MRN it had just pseudonymized at PID-3, in the same message, while the
determiner's inventory labelled it OTHER_UNIQUE_ID.

  - PV1-19 now carries routeByTypeCode, resolving its category per repetition
    from CX-5 exactly as PID-3 does. An MR-typed visit number is pseudonymized
    to the SAME surrogate as PID-3; AN/SS/MA are transformed likewise; only a
    VN-typed or untyped value is the encounter identifier and retainable.
  - LIMITED_DATA_SET_DIRECT_IDENTIFIERS encodes the sixteen direct identifiers
    as data, so the citation is enforced rather than restated in prose. Exactly
    two of the eighteen categories are retainable: DATES and (R).
  - The refusal is defence in depth: the extractor will not mark such a locus,
    AND the engine refuses it anyway via isRetainableCategory().

F3 (major, criterion 7) -- the engine retained on the locus marker alone and
never read options.retainedLoci, so DeidOptions.retainedLoci's own JSDoc ("absent
or empty retains nothing") was false at the layer it was declared on, and the
class contract was unenforceable because the marker was a class-less boolean.
GenericLocus.retainedByPolicy?: boolean becomes retention?: RetainedLocusClass,
and handleLocus now needs THREE keys to line up: an adapter proposal, the
configured options naming that class, and a retainable category.

F4 (major) -- a hand-built bag pairing the reserved `safe-harbor` label with a
retention set returned all seven loci while the report printed
`Policy: safe-harbor`. assertRetentionContract() makes that a fatal
DEID_POLICY_INVALID, the retention analogue of the existing date-shift label
guard, on the one route no profile-level check can see.

F1 (major, criterion 8) -- the corrected honesty page replaced a true broad
limitation with a false narrow enumeration: it named SPM-17 and PV1-7/8/OBR-16
as though complete, when a routine message carries 15 more unrecorded
full-precision timestamps (EVN, PV2, PR1, RXA, RXD, FT1, TXA, SPM). Criterion 8
tests exactly this, and a new false claim is worse than the old one. The
enumeration is DELETED and replaced by the true class statement in
limitations.md, README.md, guides-hl7.md, src/hl7/index.ts, src/hl7/retain.ts
and the changeset.

F5 (minor) -- the inventory printed a kept year and a kept 14-digit timestamp
identically. RetainedQuasiIdentifier now carries its transform, and the
rendering says "coarse residual" or "whole value kept".

F6 (minor) -- that the retention classes are HL7-only is now stated in the
retention module and the changeset rather than left to be discovered.

F7 and F8 are PRE-EXISTING (they reproduce on f8b7f56) and are reported as
backlog lines, not fixed here.

verify.sh green: 621 tests, typecheck, lint, format:check, check, phi-scan,
check-no-internal-refs, coverage, build, attw, smoke.
…cked

conformance-refuter pass 2 returned VERDICT: NOT REFUTED, with one INTRODUCED
minor (N2) it flagged as worth fixing before merge: four published lines still
said PV1-19 is "blocked as the (R) catch-all" and none mentioned the type-code
routing the remedy added. After that remedy an MR/AN/SS-typed PV1-19 is
PSEUDONYMIZED or REDACTED as the identifier it really is, under BOTH profiles,
and the limited-data-set line "keeps the visit number unchanged" lacked the
caveat that this is only ever true of a VN-typed or untyped one.

The defect under-states protection rather than over-stating it, so it is not the
class of claim defect that gated pass 1 -- but a published surface that
describes a mechanism the code no longer has is a claim defect either way.
Corrected in docs-content/limitations.md, docs-content/guides-hl7.md,
README.md and src/hl7/index.ts. The changeset already described the routing in
full and is unchanged.

Not fixed here, held as backlog lines because they reproduce on f8b7f56:
  - N1: the id-number write-back clears CX.1 only, so a fully-populated PV1-19
    keeps CX.6 assigning facility and the CX.7/CX.8 effective/expiration dates
    while the manifest calls the row `blocked`. Same shape as the existing
    PID-3 behaviour for an SS-typed repetition.
  - F2 residual: an UNTYPED PV1-19 carrying an MRN is still retainable under
    limited-data-set. The wire gives no signal, and no surface claims otherwise.
  - F7: ~15 unrecorded full-precision timestamps across EVN, PV2, PR1, RXA,
    RXD, FT1, TXA, SPM, plus a DOB riding through as OBX-5 with a DTM type.
  - F8: under limited-data-set, PID-7 is date-shifted while the retained
    encounter dates are not, so intervals are not preserved across that line.

verify.sh green. Refuter passes spent: 2 of 4.
@NSchatz
NSchatz merged commit ba257b9 into main Aug 11, 2026
10 checks passed
@NSchatz
NSchatz deleted the deid-encounter-loci branch August 11, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant